Record and Playback Mouse and Keyboard Events - Felix John COLIBRI.

• abstract : record and playback all mouse and keyboard messages : all windows messages are saved in a list (or a file) and can be replayed on the spot or later. A must for debugging VCL components, event logging, keyboard macro, computer based training. Includes a readable format which can be used for SendKey / SendClicks replays, simulations or unit test.
  
• key words : record - playback - Windows Messages - windows hooks - wh_JournalRecord - wh_JournalPlayback - SetWindowsHookEx - tEventMsg - CallNextHookEx - UnhookWindowsHookEx - ClientToScreen - virtual key codes
  
• software used : Windows XP Home, Delphi 6
  
• hardware used : Pentium 2.800Mhz, 512 M memory, 140 G hard disc
  
• scope : Delphi 1 to 2006, Turbo Delphi for Windows, Kylix
  Delphi 5, 6, 7, 8 Delphi 2005, 2006, Turbo Delphi, Turbo 2007, Rad Studio 2007 to 2009, Delphi XE
  
• level : Delphi developer
  
• plan :
  
  • Record and Playback for Debugging
    
  • wh_JournalRecord and wh_JournalPlayback Windows Hook
    
  • Delphi Mouse and Keyboard event Journal and Replay
    
  • Download the Sources
    

1 - Record and Playback for Debugging

So we decided to bite the bullet and build a reasonable record / playback system which could be used in a non invasive way during our test.

The specification was

• record and replay keyboard events
  
• record and replay mouse events. Since our grid does not use mouse movement (we do not implement "onMouseOver" type of feedback, and resizing the columns was not under scrutiny), recording mouse moves was not included. But this can be easily reintroduced (commenting out an If)
  
• be able to visualize the recorded messages (mainly for understanding the recording mechanism)
  
• to be able to replay as many time as desired the recorded messages
  
• be able to save, and reload the messages later, with the same application, but not necessarily at the same screen position
  
• be able to add messages manually, using a textual representation
  

2 - wh_JournalRecord and wh_JournalPlayback Windows Hooks

2.1 - Windows Hooks

Hooks are a basic Windows functionality which enables us to ask Windows to call one of our procedure. The same old technique already used in Interrupt redirection

• we make a call to tell Windows where our callback is
  
• the callback is inserted in a (possibly empty) callback queue
  
• when the target event happens, Windows calls our procedure. We then
  
  • perform whatever tasks we desire
    
  • we usually call the next callback in the queue
    

2.2 - wh_JournalRecord Windows Journaling hook

Where

• my_journal_record_callback the callback we have to write
  
• hInstance in the .EXE instance
  
• my_hook_handle will be used to later remove the hook from the chain
  

• every Window application has a "message loop" :

  While GetMessage Do Begin   TranslateMessage;   DispatchMessage; End;

  For Delphi, this is nested deep inside Application.Run (in the .DPR)

• the user types on the keyboard and moves the mouse around
  
• the messages are stored in a message queue
  
• meanwhile, the While loop runs until we quit the program, and whenever there is some message in the queue, it is removed from the queue and the corresponding message handler, if created, is called

Our callback will have the following header:

where

• p_hook_code is a "hook code", with the following values
  
  • HC_SYSMODALOFF : a system-modal dialog box has been destroyed. The hook procedure must resume recording.
    
  • HC_SYSMODALON : a system-modal dialog box is being displayed. Until the dialog box is destroyed, the hook procedure must stop recording.
    
  • HC_ACTION : the other keyboard and mouse events.
    
• p_w_param is not used.
  
• p_l_param is a pointer to the message information
  
• the return value is not used
  

The tEventMsg, defined in WINDOWS.PAS has the following definition:

where

• message is the message code (wm_MouseDown, wm_KeyUp etc)
  
• paramL and paramH are parameters depending on the type of event
  
• time is the tick count of the event (allowing to detect double clicks)
  
• hwnd is the handle of the window to which the message was posted. It allows Windows to set the focus
  

• for the mouse events
  
  • the low word of paramL contains the SCREEN x position
    
  • the low word of paramH contains the SCREEN y position
    
• for keyboard events
  
  • only wm_KeyDown and wm_KeyUp are captured (the wm_Char is generated by the TranslateMessage in the message loop)
    
  • the virtual key code is in the lo byte of paramL
    

• to store the tEventMsg somewhere, to be in a position to fetch it back for replay
  
• to call the next possible hook callback. This is done inside our callback by calling:

  Result:= CallNextHookEx(my_hook_handle, p_hook_code,      p_w_param, p_l_param);

• we are supposed to handle the vk_Cancel virtual key code, which is triggered by typing Ctrl+ Break, and we should terminate the journaling when we receive this key combination.

  We can also specify other key combinations in order to stop recording. In our application, we trigger the end by clicking on an tButton. We also noticed that within the IDE, Ctrl Alt Del also stops the journaling
  

Finally, note that

• the hook is system wide. It cannot be used as thread specific
  
• the callback is always handled in the context of our thread
  

• the tEventMsg message parameters are the "raw unprocessed" windows message informations.
  
• the normal driver processing will then take into account the national keyboard, the currently typed keys like Shift, Alt etc to qualify the raw parameters. vk_A might become "a", "A", "â" etc. Similarily the mouse events do not contain any p_button parameter or shift states or other values we usually find in OnKeyDown or OnMouseDown Delphi events.
  
• so the tMessageEvent are the raw data before those processing. We store those, and during playback, the raw parameters will be reinjected just before the driver processing, thus producing the very nice event parameters we expect in Delphi
  

2.3 - wh_JournalPlayback Windows Journaling hook

• first, we install the replay hook by calling

  my_hook_handle:= SetWindowsHookEx(wh_JournalPlayback,      my_journal_playback_callback, hInstance, 0);

  Once the replay hook is installed, the GetMessage will call our callback to get the messages from our message store, instead of waiting for the user events:

• the replaying callback has the following header:

  Function journal_playback_callback(p_hook_code: integer;      p_w_param, p_l_param: Longint): Longint; Stdcall;

  and:

  • p_hook_code is used to specify the type of action. Possible values are:
    
    • HC_NOREMOVE : the recording application had called the PeekMessage function with wRemoveMsg set to PM_NOREMOVE, indicating that the message is not removed from the message queue after PeekMessage processing.

    • HC_SYSMODALON : A system-modal dialog box is being displayed. Until the dialog box is destroyed, the hook procedure must stop playing back messages.
      
    • HC_SYSMODALOFF : A system-modal dialog box has been destroyed. The hook procedure must resume playing back the messages.

    • HC_SKIP : The hook procedure must prepare to copy the next mouse or keyboard message to the tEventMsg structure pointed to by lParam. Upon receiving the HC_GETNEXT code, the hook procedure must copy the message to the structure.

      If there are no more messages in our store, we call:

      UnhookWindowsHookEx(my_hook_handle);

    • HC_GETNEXT : the callback must copy the current mouse or keyboard message to the tEventMsg structure pointed to by p_l_param
      
  • p_w_param : not used
    
  • p_l_param : A pointer to an tEventMsg structure that represents a message being processed by the hook procedure. This parameter is valid only when the code parameter is HC_GETNEXT.

  • the result of the function specifies the amount of time, in clock ticks, that the system should wait before replaying the message:
    • This value can be computed by calculating the difference between the time attribute in the current and previous recorded tEventMsg. It can be 0.
      
    • To process the message immediately, the return value should be zero.
      
    • this value is used only if the hook code is HC_GETNEXT, otherwise, it is ignored.
      
  Our basic tasks are :
  • to handle HC_SKIP for initializing a tEventMsg variable
    
  • to handle HC_GETNEXT, where we initialize p_l_param to point to this variable. Eventually we tell how much tick we should wait
    
  • to call the possible next hook playback, with:

    Result:= CallNextHookEx(my_hook_handle,      p_hook_code,  p_w_param, p_l_param);

    
    

• to replay the same tEventMessage several time, we simply keep the same value in our tEventMessage (HC_SKIP does not grab the next message, but keeps the current one for some iteration).
  
• to sleep between the replay of a message
  
  • in a first callback, in HC_GETNEXT we return the wait tick count.
    
  • after the pause, Windows calls back our procedure, and in HC_GetNext we now should return 0
    
  This could be used to slow the replay down
  
• the callback is always handled in the context of our thread (the one which installed the wh_JournalReplay). It can be in our project, or in a Library (a .DLL)
  
• If the user presses Ctrl+Esc or Ctrl+Alt+Del during journal playback, the system stops the playback, unhooks the journal playback procedure, and posts a wm_CancelJournal message to the journaling application.
  
• if the hook procedure returns a message in the range wm_KeyFirst to wm_KeyLast, the following conditions apply:
  • tEventMsg.paramL specifies the virtual key code of the key that was pressed.
    
  • tEventMsg.paramH specifies the scan code (the keyboard touch number).
    
  • there's no way to specify a repeat count. The event is always taken to represent one key event.
    

3 - Delphi Mouse and Keyboard event Journal and Replay

3.1 - Overall architecture

• the c_message_list is a tStringList container which stores the tEventMessages (in memory or disc)
  
• to be able to replay the messages from a disc file, we must be able to set the window handles of the controls in the application which reloads the messages. Our c_win_control_list is in charge of mapping the control names to the Windows handles
  
• our c_record_playback_message Class is in charge of launching the record and playback, and contains the two callbacks
  
• finally a c_message_list_parser is able to parse a textual tEventMsg representation
  

3.2 - The UML class diagram

3.3 - The tWinControl list

The definitions are:

The structure is loaded by a recursive procedure using the Controls array:

and the f_screen_x and f_scree_y are used to compute the current SCREEN position of the top left corner of any control on our form :

3.4 - The message list

And:

• the t_message has the same structure as the tEventMsg
  
• the t_pt_message pointer is used to cast the p_l_param in the record and replay callback
  
• t_message_with_control_name is a t_message along with the control name.

  Basically the t_message contains the window handle at the time of recording. If we replay during the same run this window handle is valid. However, if we save the messages on disc and reload them during another execution, the handle will no longer valid. We must somehow change the handles to match the handles of the actual controls.

  Therefore

  • when we save the messages, we naturally save the t_message parameters, and also the control name (say 'Edit3', or 'PaintBox5'.
    This name is looked up using the c_win_control_list. This list is built when we create the c_message_list, and used when we save the message list
    
  • whenever we reload the message from a file
    
    • we create a c_message_list, and rebuild the c_win_control_list with the current controls of the application
      
    • for each message read from the disc, we read the control name ('Panel18') and compute the window handle of the controls in this application. This is only required for mouse messages. Hence the c_message.f_is_mouse_message function
      
• the c_message_list
  
  • has a Constructor with a tWinControl parameter. We call the Constructor with the main tForm object, and this object is used to recursively build the c_win_control control / handle map
    
  • we have several functions for appending messages to our list
    
  • the recorded t_message.m_time contains the tick count at the time of the recording.

    When we replay the message later, we might want do use the actual tick count.

    Therefore:

    • when we start the recording, we initialize the m_start_tick_count value (using a call to GetTickCount)
      
    • when we stop the recording, we compute a relative tick count, subtracting m_start_tick_count from all time values
      
    • when we replay, we initialize m_start_tick_count, and add this value back to the relative value saved in the message store
      
      
  • the same goes on for the mouse X, Y positions
    
    • at recording time, the values are relative to the SCREEN (the recording is SYSTEM WIDE)
      
    • when we reload the message list from the file, maybe a couple of days later, the window might not have the same screen position, or we might even have moved some controls away from one another (using splitters or tAlign and Anchor properties). So the screen position would be wrong, and all the mouse messages might not work properly (usually nothing happens, but we might as well click on some other button)
      
    • therefore, when we stop recording, we decrement the mouse values by the left top values of the win control. This is performed using the c_win_control_list. So the X, Y values are the control-relative mouse position
      
    • at replay time, the c_win_control_list has been initialized with the current control positions (the same as during the recording, or the values recomputed when we create the message list).
      The screen control offset are then added back to each message before replay
      
  • removing the absolute time and position values is performed in the compute_relative_tick_count_and_mouse_xy procedure
    
  To recap,
  • after the recording ends, the relative time and mouse positions are computed
    
  • the message list is saved after recording
    
  • therefore, after recording, the in-memory or the possibly saved message list always contain relative value
    
  • at replay time, the absolute values are recomputed, using the current tick count as well as the current control positions
    

• the time value seems irrelevant. Computing the "nearly" actual time by adding back GetTickCount is not necessary. Running the list with relative times, or even with 0 time values (as we do for the manually crafted messages below) has the same result. However, we could use the relative values to inject some delays between the messages (return a non zero values from the callback)
  

3.5 - The record and playback Class

and:

• the recorded message list is managed by c_message_list
  
• when we install the hook (a call to start_recording), the returned handle is saved in m_hook_handle. At the same time, m_is_recording is toggled to true to avoid recursive recording
  
• before replaying, we initialize m_replay_start_index and m_replay_end_index:
  • by default the values are 0 and the message count- 1
    
  • however we might want to skip some initial messages (while tracking a bug), and usually we remove the last two messages which are the click on the "stop_recording" button
    
• when we call start_replay
  
  • the current message index m_replay_index is initialized to m_replay_start_index, and m_is_playing is initialized to True
    
  • each time hc_Skip is reached
    
    • if m_replay_index is greater than m_replay_end_index, we call stop_playback
      
    • if not we get the the next message (make a copy of the current message and adjust the relative time and position values) and increment the current index
      
• m_on_notify is a callback event that we used to display the current message being replayed
  

And our replay callback is :

Finally, we compute the next message to replay with:

And

• before installing the journal replay hook, we call get_next_message(0)
  
• in each playback call, we increment the index and call this method again
  

• the message used for playback is contained in a separate variable. We copy the values of the current message in this structure and adjust this variable to get the absolute time and screen position
  
• it was a design decision to recompute the values for each run in a separate variable instead of directly modifying the values in the message list.
  This former solution required using booleans to remember whether the values were absolute or relative, and quickly became a mess. And copying 20 bytes today is not such a big deal
  

Our message list contains all kind of display function that you can call at any time (while recording, after the recording has stopped, after reloading the saved messages, while replaying etc)

3.6 - Forging messages

• for keyboard message, we simply convert some textual information (say "Return" or "A" into the binary wm_code, wParam and lParam
  
• for mouse messages, we have to build the wm_code, then screen position and the window handle
  

• for keyboard messages
  
  • the ascii character (say "a", or "b", or "c")
    
  • for non printable key, the virtual key code (say "vk_Return", "vk_F2", "vk_Shift" etc, with an additional incremental delay parameter measured in ticks. To avoid confusing letters "vk_Return" with "v", "k" "_", we enclose the coding in an "improbable" character. For now we chose the greek letter MU

    The coding of vk_Home with a 400 tick delay would look like

        µvk_home,400µ
    

  Our parsing unit contains a function

  Const k_virtual_key_code_separator= 'µ'; Function f_relative_virtual_key_code_text(p_virtual_key_code: Word;     p_delta_time: Cardinal): String;     // -- format "µvk_tab,R_tickµ"   Begin     Result:= k_virtual_key_code_separator         + LowerCase(f_virtual_key_code_name(p_virtual_key_code))         + ','         + IntToStr(p_delta_time)         + k_virtual_key_code_separator         ;   End; // f_virtual_key_code_text

• for mouse messages, we used special "vk_" texts and the X, Y position, as well as a

  Our encoding function is:

  Const k_left_mouse_down_name= 'vk_left_mouse_down';       k_left_mouse_up_name= 'vk_left_mouse_up';       k_right_mouse_down_name= 'vk_right_mouse_down';       k_right_mouse_up_name= 'vk_right_mouse_up'; Function f_relative_mouse_text(p_mouse_message: String;     p_delta_x, p_delta_y, p_delta_time: Integer; p_control_name: String): String;     // -- format "µvk_left_mouse_down,R_NNN,R_NNN,R_tick,NAMEµ"     // -- like "µvk_left_mouse_downµ200,100,4000,Edit4"   Begin     Result:= Format(k_virtual_key_code_separator             + '%s,%d,%d,%d,%s'+ k_virtual_key_code_separator,         [p_mouse_message, p_delta_x, p_delta_y, p_delta_time, p_control_name]);   End; // f_relative_mouse_text

  
  

• selects Edit3
  
• types "a"
  
• goes to the next control (assuming it is another edit), by typing "Tab"
  
• goes at the start of the edit (avoiding to overwrite the AutoSelected Text) by typing "Home"
  
• typing "b"
  

which will create the following text (line break added) :

This line is parsed by calling parse_relative_message_string, and the resulting c_message_list can be replayed, saved, reloaded etc, like any other recorded message.

The detail of the parsing in included in the downloadable .ZIP

3.7 - The Delphi Journaling Hook demo

	we started the app, clicked the "record_" toggle once, and clicked on Edit3 types "a" Tab End "b"
	Here is the snapshot :
	to stop the recording, we clicked the "records_" toggle a second time
	to display the message, we clicked "display_message_list_":
	this is the message display :
	we clicked "playback_"
	here is the result: notice that the letters have been double ("aa" and "bb") the caret is after the second "b"
	you can replay again. Make sure you click on "playback_" again, since the focus is now in Edit4. You can save, reload another day etc

To demonstrate the textual message feature

	click "generate and parse"
	here is the display:
	you can display, save, playback the message

Just for the record, writing this little app took us 2 days, plus one for the article. Enjoy !

4 - Improvements

• provide for a multi-form journaling. This would require saving the form name along with the control name in the win control list
  
• use a more concise textual representation for messages (like "~" for Ctrl, "@" for Alt etc)
  
• implement repeat or delay features
  
• add automatic unhook procedures, at least if we quit the project
  
• our tWinControl structure could have been optimized (sorted, for instance), and maybe before replay we could include in each c_message a reference to the tWinControl to avoid lookups
  

5 - Download the Sources

• mouse_and_keyboard_record_and_playback.zip: the project with
  
  • the win control list
    
  • the message list
    
  • the record / playback manager
    
  • the textual message parser
    mouse_and_keyboard_record_and_playback.zip creating
    
  Size : (38 K)
  

• the main program (.DPR, .DOF, .RES), the main form (.PAS, .DFM), and any other auxiliary form
  
• any .TXT for parameters, samples, test data
  
• all units (.PAS) for units
  

• are self-contained: you will not need any other product (unless expressly mentioned).
  
• for Delphi 6 projects, can be used from any folder (the pathes are RELATIVE)
  
• will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path creation etc).
  

• create or select any folder of your choice
  
• unzip the downloaded file
  
• using Delphi, compile and execute
  

The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lass etc. This notation is presented in the Alsacian Notation paper.
The .ZIP file(s) contain:

• the main program (.DPROJ, .DPR, .RES), the main form (.PAS, .ASPX), and any other auxiliary form or files
  
• any .TXT for parameters, samples, test data
  
• all units (.PAS .ASPX and other) for units
  

• are self-contained: you will not need any other product (unless expressly mentioned).
  
• will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path outside from the container path creation etc).
  

• create or select any folder of your choice.
  
• unzip the downloaded file
  
• using Delphi, compile and execute
  

The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lass etc. This notation is presented in the Alsacian Notation paper.

As usual:

• please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will be helpful for other readers
  
• we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
  
• or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button

  Name :
  E-mail :
  Comments * :

• and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate: the more traffic and Google references we get, the more articles we will write.
  

6 - References

• Hooks Overview
  
• JournalRecordProc Callback Function
  
• JournalPlaybackProc Callback Function
  

7 - The author